Adaptive interface for screen-based interactions

ABSTRACT

Systems and methods for customizing an output based on user data are described herein. An example method for customizing an output based on user data may commence with capturing, by at least one sensor, the user data. The method may continue with analyzing, by at least one computing resource, the user data received from the at least one sensor. The method may further include continuously customizing, by an adaptive interface, output data using at least one machine learning technique based on the analysis of the user data. The customized output data may be intended to elicit a personalized change.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present utility patent application is related to and claims thepriority benefit under 35 U.S.C. 119(e) of U.S. provisional patentapplication No. 62/605,179, filed on Aug. 4, 2017, titled “Adaptiveinterface for screen-based interaction,” and U.S. provisional patentapplication No. 62/541,899, filed on Aug. 7, 2017, titled “Adaptiveinterface with machine learning and biofeedback for screen and/oraudio-based interactions. Machine learning processes sensor data, andoutputs dynamic screen and audio-based interaction, e.g., graphical userinterface (GUI), computer generated images (CGI), visual and/or audioenvironment, and user experience design (UX). Screen and/or audio workenvironment and interactions are modified and adapted to the data fromthe user using machine learning based on input from sensor.” Thedisclosures of these related provisional applications are incorporatedherein by reference for all purposes to the extent that such subjectmatter is not inconsistent herewith or limiting hereof.

TECHNICAL FIELD

The present disclosure relates generally to data processing and, moreparticularly, to customizing output data on an electronic deviceassociated with a user based on biological data of the user.

BACKGROUND

Conventionally, people use digital devices, such as smartphones,tablets, and laptops, in many environments with different lightingconditions, e.g., indoors in daylight, indoors in artificial light,outdoors in clear weather, outdoors in cloudy weather, and the like. Thedigital devices may be configured to automatically adjust displayparameters to suit the environmental conditions and the content a useris currently viewing. In other words, a digital device may have ‘anadaptive display’ feature that may enable the digital device toautomatically adjust a color range, contrast, and sharpness of a displayaccording to the current usage of the digital device by the user. Thedigital device may sense the environmental conditions, determine thetype of content the user is viewing, determine a particular applicationthe user is using, and analyze all collected data to select parametersfor optimizing the viewing experience of the user.

Additionally, according to scientific studies, exposure to blue light ofthe visible light spectrum was found to have an impact on health of aperson by contributing to eye strain. Blue light also was determined tobe important in regulating sleep/wake cycles of a body of the person.The display screens of smartphones, computers, laptops, and otherdigital devices are sources of significant amounts of blue light.Therefore, some conventional digital devices are configured to adapt thecolor range of a display by activating blue light filtering at night orat time intervals selected by the user to reduce amounts of blue lightemitted by the screens.

However, although displays of digital devices can be adjusted based onparticular environmental parameters collected by the digital devices andcurrent settings of the digital devices, conventional digital devices donot analyze the current physiological state of the user when adjustingthe parameters of the display. Therefore, the adjusted parameters of thedigital device may be irrelevant to physiological parameters of theuser.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Provided are computer-implemented systems and methods for customizingoutput based on user data. In some example embodiments, a machinelearning system for customizing output based on user data may include atleast one sensor, at least one computing resource, and an adaptiveinterface. The at least one sensor may be configured to capture the userdata. The at least one computing resource may be configured to analyzethe user data received from the at least one sensor. The adaptiveinterface may be configured to continuously customize output data usingat least one machine learning technique based on the analysis of theuser data. The customized output data may be intended to elicit apersonalized change.

In some example embodiments, a method for customizing an output based onuser data may commence with capturing, by at least one sensor, the userdata. The method may continue with analyzing, by at least one computingresource, the user data received from the at least one sensor. Themethod may further include continuously customizing, by an adaptiveinterface, output data using at least one machine learning techniquebased on the analysis of the user data. The customized output data maybe intended to elicit a personalized change.

Additional objects, advantages, and novel features will be set forth inpart in the detailed description section of this disclosure, whichfollows, and in part will become apparent to those skilled in the artupon examination of this specification and the accompanying drawings ormay be learned by production or operation of the example embodiments.The objects and advantages of the concepts may be realized and attainedby means of the methodologies, instrumentalities, and combinationsparticularly pointed out in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example and not limitation in thefigures of the accompanying drawings, in which like references indicatesimilar elements and, in which:

FIG. 1 illustrates an environment within which systems and methods forcustomizing output based on user data can be implemented, in accordancewith some embodiments.

FIG. 2 is a block diagram showing various modules of a machine learningsystem for customizing output based on user data, in accordance withcertain embodiments.

FIG. 3 is a flow chart illustrating a method for customizing outputbased on user data, in accordance with some example embodiments.

FIG. 4 illustrates a further example environment within which systemsand methods for customizing an output based on user data may beimplemented, in accordance with some example embodiments.

FIG. 5 illustrates a further environment within which systems andmethods for customizing output based on user data can be implemented, inaccordance with some example embodiments.

FIG. 6 is a schematic diagram that illustrates operations performed bycomponents of a machine learning system for customizing output based onuser data, in accordance with some example embodiments.

FIG. 7 is a schematic diagram that illustrates operations performed byan adaptive interface to customize output on a user device based on userdata, in accordance with some example embodiments.

FIG. 8 is a flow chart illustrating customization of output data of auser device based on user data, in accordance with some exampleembodiments.

FIG. 9 is a schematic diagram showing customization of output data on auser device based on biological data of a user, according to an exampleembodiment.

FIG. 10 is a schematic diagram illustrating processing data from asensor using machine learning processing, according to an exampleembodiment.

FIG. 11 is a flow chart illustrating continuous customization of outputbased on user data, according to an example embodiment.

FIG. 12 is a schematic diagram showing operations performed by anadaptive interface to continuously customize output data using machinelearning techniques, according to an example embodiment.

FIG. 13 is a schematic diagram showing operations performed by anadaptive interface to continuously customize output data using machinelearning techniques, according to an example embodiment.

FIG. 14 is a block diagram illustrating continuous personalization of abrightness level on a user device based on data related to respirationor heart rate of a user, according to an example embodiment.

FIG. 15 is a block diagram illustrating continuous personalization of avolume level on a user device based on data related to respiration orheart rate of a user, according to an example embodiment.

FIG. 16 is a block diagram illustrating continuous personalization of anodorant level on a user device based on data related to respiration orheart rate of a user, according to an example embodiment.

FIG. 17 is a schematic diagram showing a user interface of a mobiledevice customized by a machine learning system for customizing outputbased on user data, according to an example embodiment.

FIG. 18 is a schematic diagram showing output data of headphonescustomized by a machine learning system for customizing output based onuser data, according to an example embodiment.

FIG. 19 is a schematic diagram showing output data of an artificialolfactory device customized by a machine learning system for customizingoutput based on user data, according to an example embodiment.

FIG. 20 is a schematic diagram showing customizing output of a userdevice based on user data captured by a digital camera of the userdevice, according to an example embodiment.

FIG. 21 is a schematic diagram showing an analysis of captured user databy an adaptive interface, according to an example embodiment.

FIG. 22 is a schematic diagram showing output data continuously adaptedby an adaptive interface, according to an example embodiment.

FIG. 23 is a flow chart showing a method for customizing output based onuser data, according to an example embodiment.

FIG. 24 shows a computing system that can be used to implement a methodfor customizing output based on user data, according to an exampleembodiment.

DETAILED DESCRIPTION

The following detailed description includes references to theaccompanying drawings, which form a part of the detailed description.The drawings show illustrations in accordance with exemplaryembodiments. These exemplary embodiments, which are also referred toherein as “examples,” are described in enough detail to enable thoseskilled in the art to practice the present subject matter. Theembodiments can be combined, other embodiments can be utilized, orstructural, logical, and electrical changes can be made withoutdeparting from the scope of what is claimed. The following detaileddescription is, therefore, not to be taken in a limiting sense, and thescope is defined by the appended claims and their equivalents.

The present disclosure provides methods and systems for customizingoutput based on user data. The system for customizing output based onuser data of the present disclosure may continuously customize an outputprovided by a digital device associated with a user to elicit apersonalized change of biological parameters of the user. Specifically,the system may collect user data, such as biological data of the user orother users, historical data of the user or other users, ambient data,and the like. The biological data may include data related to physicalparameters of the user, e.g., a heart rate, body temperature, a bloodoxidation level, presence of a substance in a blood, a blood glucoselevel, and the like. The user data may be collected by sensors affixedto the user, such as a heart-rate monitor; sensor located in proximityto the user, such as a thermal imaging camera and a digital camera;sensors embedded into the digital device of the user; and the like. Thesystem may analyze the collected user data.

The system includes an adaptive interface unit, also referred herein toas an adaptive interface, that uses the results of the analysis tocontinuously customize output data on the digital device of the user,also referred to herein as a user device. The adaptive interface appliesmachine learning techniques to process the results of analysis of thecollected user data and select changes to a graphics output and/or anaudio output on the user device to cause the change in biological dataof the user. Specifically, the adaptive interface may continuouslyanalyze the relationship between the biological data of the user, suchas a heart rate, and the graphics and audio the user experiences whenusing the digital device. The results of continuous analysis ofdependencies between the biological data of the user and the output dataprovided to the user on the user device may be stored in a database in aform of historic data associated with the user. Additionally, data ondependencies between biological data of a plurality of users and outputdata on digital devices of the plurality of users may be stored in thedatabase (for example, in a form of historic data associated with theplurality of users).

Therefore, the adaptive interface may determine, based on the analysisof the collected user data, that the user has an increased heart rate atthe current moment of time, and adapt the output data on the interfaceof the digital device to elicit reduction of the heart rate of the user.For example, the adaptive interface may determine, based on the historicdata, that the heart rate of the user typically changes in response tochange of the volume of the audio output and brightness of the videooutput. Based on such determination, the adaptive interface may reducethe volume of the audio output and decrease the brightness of a displayof the user device to cause the reduction of the heart rate of the user.

Thus, the adaptive interface may relate to bio-adaptive technology andmay perform the adaptation of an output of the user device based onbiological parameters of the user. The adaptation of the output of theuser device may be directed to eliciting the change of the biologicalparameters of the user in case the biological parameters of the user donot correspond to predetermined ranges or values.

FIG. 1 illustrates an environment 100 within which systems and methodsfor customizing output based on user data can be implemented, inaccordance with some embodiments. The environment 100 may include afrontend 101 and a backend 103. The frontend 101 may include a sensor106 and a user device 104 associated with a user 102. The backend 103may include a machine learning system 200 for customizing output basedon user data (also referred to as a system 200), a server 108, a datanetwork shown as a network 110 (e.g., the Internet or a computingcloud), and a database 112. The user device 104, the system 200, theserver 108, the sensor 106, and the database 112 may be connected viathe network 110.

The user 102 may be associated with the user device 104. The user device104 may include a smartphone 114, a laptop 116, headphones 118, aretinal implant 120, an artificial olfaction device 122, and so forth.In an example embodiment, the artificial olfaction device 122 mayinclude an electronic system operating as an electronic nose of the user102.

The network 110 may include a computing cloud, the Internet, or anyother network capable of communicating data between devices. Suitablenetworks may include or interface with any one or more of, for instance,a local intranet, a corporate data network, a data center network, ahome data network, a Personal Area Network, a Local Area Network (LAN),a Wide Area Network (WAN), a Metropolitan Area Network, a virtualprivate network, a storage area network, a frame relay connection, anAdvanced Intelligent Network connection, a synchronous optical networkconnection, a digital T1, T3, E1 or E3 line, Digital Data Serviceconnection, Digital Subscriber Line connection, an Ethernet connection,an Integrated Services Digital Network line, a dial-up port such as aV.90, V.34 or V.34bis analog modem connection, a cable modem, anAsynchronous Transfer Mode connection, or a Fiber Distributed DataInterface or Copper Distributed Data Interface connection. Furthermore,communications may also include links to any of a variety of wirelessnetworks, including Wireless Application Protocol, General Packet RadioService, Global System for Mobile Communication, Code Division MultipleAccess or Time Division Multiple Access, cellular phone networks, GlobalPositioning System, cellular digital packet data, Research in Motion,Limited duplex paging network, Bluetooth radio, or an IEEE 802.11-basedradio frequency network. The data network can further include orinterface with any one or more of a Recommended Standard 232 (RS-232)serial connection, an IEEE-1394 (FireWire) connection, a Fiber Channelconnection, an IrDA (infrared) port, a Small Computer Systems Interfaceconnection, a Universal Serial Bus (USB) connection or other wired orwireless, digital or analog interface or connection, mesh or Digi®networking.

The sensor 106 may be affixed to any body part of the user 102.Alternatively, the sensor 106 may be located in proximity to the user102. In a further example embodiment, the sensor 106 may be integratedinto the user device 104. The sensor 106 may include a biological sensor(e.g., a heart-rate monitor), a thermal imaging camera, a breath sensor,a radar sensor, and the like. The sensor 106 may collect user data 124and provide the collected user data 124 to the user device 104. The userdevice 104 may provide the user data 124 to the system 200.

The system 200 may be running on the user device 104 or in the computingcloud. The system 200 may have an access to output data reproduced bythe user device 104, such as graphics and audio. The system 200 mayinclude a computing resource 204 and an adaptive interface 206. Thecomputing resource 204 of the system 200 may analyze the user data 124.The adaptive interface 206 may apply machine learning techniques 126 tothe results of the analysis to customize the output data of the userdevice 104 so as to cause changing of the biological data of the user102. In an example embodiment, the adaptive interface 206 may include acombination of sensors, machine learning algorithms, processing units,and computing resources. The adaptive interface 206 may reside in theuser device 104 or remotely to the user device, e.g., in the computingcloud.

The adaptive interface 206 may also send the data obtained based onprocessing of the user data using machine learning algorithms to theserver 108 to update the data of an application running on the userdevice 102. The server 108 can include at least one controller and/or atleast one processor. An alternate implementation of the server 108 caninclude an application or software running on the user device 104. Theserver 108 can update and improve code associated with the applicationusing data associated with the plurality of individual users. The server108 can then send the updated output data associated with theapplication to the adaptive interface 106 via the network 110 forfurther displaying on the user device 104.

FIG. 2 is a block diagram showing various modules of a machine learningsystem 200 for customizing output based on user data, in accordance withcertain embodiments. The system 200 may include at least one sensorshown as sensor 202, at least one computing resource shown as acomputing resource 204, an adaptive interface unit shown as an adaptiveinterface 206, and optionally a database 208. The database 208 mayinclude computer-readable instructions for execution by the computingresource 204 and the adaptive interface 206. In an example embodiment,each of the computing resource 204 and the adaptive interface 206 may beimplemented as one or more processors. The processor may include aprogrammable processor, such as a microcontroller, a central processingunit (CPU), and so forth. In other embodiments, the processor mayinclude an application-specific integrated circuit or programmable logicarray, such as a field programmable gate array, designed to implementthe functions performed by the system 200. In various embodiments, thesystem 200 may be installed on a user device or may be provided as acloud service residing in a cloud storage.

The sensor 202 may be affixed to a user, integrated into the userdevice, or located in proximity to the user. The sensor 202 may includeat least one of the following: a thermal imaging camera, a digitalcamera, a breath sensor, a depth sensor, a radar sensor, a gyroscope,and so forth. In an example embodiment, the sensor 202 may include abiological sensor. The sensor 202 may be configured to capture the userdata. The user data may include at least one of the following:biological data of a user, biological data of a plurality of users,historical data of the user, historical data of the plurality of users,ambient data, and so forth. The biological data may include at least oneof the following: a respiratory rate, a heart rate, a heart ratevariability, an electroencephalography, an electrocardiography, anelectromyography, an electrodermal activity, a mechanomyography, ahaptic interaction, a motion, a gesture, pupil movement, a biologicalanalyte, a biological structure, a microorganism, a color of skin of theuser, a blood glucose level, blood oxygenation, blood pressure, and soforth. The ambient data may be associated with at least one of thefollowing: light, heat, motion, moisture, pressure, and so forth.

The computing resource 204 may be configured to analyze the user datareceived from the sensor 202. The computing resource 204 may include atleast one of the following: an application programming interface (API),a server, a cloud computing resource, a database, a network, ablockchain, and so forth. In an example embodiment, the at least onecomputing resource that may be implemented as the user device associatedwith the user may include one of the following: a smartphone, a tabletcomputer, a phablet computer, a laptop computer, a desktop computer, anaugmented reality device, a virtual reality device, a mixed realitydevice, a retinal implant, an artificial olfaction device, headphones,an audio output device, and so forth. In a further example embodiment,the computing resource 204 may include one of a CPU, a graphicsprocessing unit (GPU), and a neural processing unit (NPU).

The adaptive interface 206 may be configured to continuously customizeoutput data of the user device using at least one machine learningtechnique based on the analysis of the user data. The at least onemachine learning technique may include one or more of the following: anartificial neural network, a convolutional neural network, a Bayesianneural network, a supervised machine learning algorithm, asemi-supervised machine learning algorithm, an unsupervised machinelearning algorithm, a reinforcement learning, a deep learning (ok toadd?) and so forth.

The customized output data may be intended to elicit a personalizedchange. The personalized change may include a change in the biologicaldata of the user. The personalized change in the user data may includeat least one of the following: a change of perception time, a change ofa respiratory rate, a change of a breathing rate, a change of a heartrate, a change of a heart rate variability, a change of a hapticinteraction, a change of an electroencephalographic signal, a change ofan electrocardiographic signal, a change of an electromyographic signal,a change of a mechanomyographic signal, a change of an electrodermalactivity, a change of a motion, a change of a gesture, a change of apupil movement, a change of a biological structure, a change of amicroorganism, a change of a color of skin of the user, a change ofblood glucose levels, a change of a blood oxygenation, a change of ablood pressure, a change of a biological analyte, a change of a stresslevel, and so forth.

The customized output data may be associated with the user device of theuser, such as a smartphone, a laptop, a retinal implant, and so forth.The customized output data may include an audio output and/or a graphicsoutput.

FIG. 3 is a flow chart illustrating a method 300 for customizing anoutput based on user data, in accordance with some example embodiments.In some embodiments, the operations may be combined, performed inparallel, or performed in a different order. The method 300 may alsoinclude additional or fewer operations than those illustrated. Themethod 300 may be performed by processing logic that may comprisehardware (e.g., decision making logic, dedicated logic, programmablelogic, and microcode), software (such as software run on ageneral-purpose computer system or a dedicated machine), or acombination of both.

The method 300 may commence at operation 302 with capturing, by at leastone sensor, the user data. The sensor may be configured to continuouslycapture the user data in real-time (e.g., when the user is awake andasleep), capture the user data during the usage of the user device bythe user, or capture the user data at predetermined times. The at leastone sensor may include a thermal imaging camera, a digital camera, abreath sensor, a depth sensor, a radar sensor, a gyroscope, a thermalimaging camera, and so forth. In an example embodiment, the at least onesensor may include a device for analyzing electronic signals emitted bya user. The method 300 may further include extracting, by the device foranalyzing electronic signals, one of a physiological parameter of theuser and an activity associated with the user.

The method 300 may continue at operation 304 with analyzing, by at leastone computing resource, the user data received from the at least onesensor. At operation 306, the method 300 may further includecontinuously customizing, by an adaptive interface, output data using atleast one machine learning technique based on the analysis of the userdata. The customized output data may be intended to elicit apersonalized change. The personalized change may include a change ofbiological parameters of the user. In an example embodiment, thecontinuous customizing of the output data may include at least one ofthe following: changing a color, playing audio-perceived stimuli,providing a haptic feedback, changing a font, changing a shape of thefont, changing a brightness, changing a contrast, changing anilluminance (e.g., changing values of lux of light), changing warmth,changing a saturation, changing a fade, changing a shadow, changing asharpness, changing a structure, generating computer images, changing atone, changing a bass, changing a volume, changing a pitch of a sound,changing a treble, changing a balance, changing a GUI, changing a UX,and so forth.

The method 300 may optionally include an operation 308, at which the atleast one computing resource may aggregate further user data associatedwith a plurality of users into federated user data. At optionaloperation 310, the at least one computing resource may analyze thefederated user data using collaborative machine learning. The method 300may further include adapting, by the at least one computing resource,the at least one machine learning technique for individual users basedon the results of the analysis of the federated user data at optionaloperation 312. In an example embodiment, the method 300 may furtherinclude continuously adapting, by the adaptive interface, a media outputbased on user interactions with the adaptive interface.

FIG. 4 illustrates a further example environment 400 in which systemsand methods for customizing an output based on user data may beimplemented, according to an example embodiment. The environment 400includes a client side, shown as a frontend 101, and a backend 103. Thefrontend 101 can include a sensor 106 and a user device 104. The backend103 can include a system 200, machine learning techniques 126, and ablockchain 402. The system 200 may include an API 404 to communicatewith the user device 104, an adaptive interface 206, and a computingresource 204. The user device 104 may include a smartphone 114, a laptop116, headphones 118, a retinal implant 120, and an artificial olfactiondevice 122, as well as a tablet computer, a phablet computer, a desktopcomputer, an augmented reality device, a virtual reality device, a mixedreality device, an audio output device, and so forth.

The sensor 106 can detect biological data of the user 102. Though twosensors 106 are shown on FIG. 4, any number of sensors, e.g., one, two,or more, may be attached to the user 102, integrated into the userdevice 104, or located in proximity to the user 102. In an exampleembodiment, the sensor 106 may detect a number of breaths per minute ofthe user 102. In other example embodiments, the sensor 106 may detectany other biological activity of the user 102, such as a heart rate,heart rate variability, electroencephalography, electromyography,mechanomyography, and so forth. In further example embodiment, thesensor 106 may be a device that analyzes electronic signals emitted bythe user 102, i.e., frequencies emitted by a body of the user 102, anddepicts the analyzed electronic signals as a biometric parameter oractivity of the user 102. In an example embodiment, the sensor 106 canbe a thermal imaging camera. The adaptive interface 206 may use deeplearning algorithms of machine learning techniques 126 to analyze theheart rate and breathing rates of the user 102 based on data collectedby the thermal imaging camera.

The sensor 106 may act as a passive sensor or an active sensor. Whenacting as a passive sensor, the sensor 106 may sense data emitted by theuser 102, such as emitted thermal wavelengths. The thermal wavelengthsmay be analyzed by the adaptive interface 206 using deep learningalgorithms of machine learning techniques 126 to determine a breathingpattern or a heart rate of the user 102. When acting as an activesensor, the sensor 106 may send towards the user 102 and receive backultrasound or radar waves. The waves received upon being reflected fromthe body of the user 102 can be analyzed by the sensor 106 to detect thephysiological state of the user 102.

The API 404 may include a Representational State Transfer (REST) API, 0API, a set of subroutine definitions, protocols, and tools for receivingdata from a server (such as a server 108 shown on FIG. 1). The API 404may provide graphics and audio on the user device 104 based on datareceived from the adaptive interface 206. The API 404 may be associatedwith one or more of the following: a web-based system, an operatingsystem, a database system, a computer hardware, and so forth.

The adaptive interface 206 may apply machine learning techniques 126including artificial neural network, convolutional neural network,Bayesian neural network, or other machine learning techniques to enablethe automatic feature learning, the machine learning inference process,and deep learning training of the adaptive interface 206. The adaptiveinterface 206 may receive data from the sensor 106, the user device 104,the API 404, a computing resource 204, and the network 110. In anexample embodiment, the computing resource 204 may be implemented as acomponent of the user device 104. In this embodiment, the adaptiveinterface 206 may communicate with and transfer data to the user device104 for data processing to use the processing units such as a GPU andCPU in the user device 104, and may apply predictive modeling andmachine learning processes.

The machine learning techniques 126 applied by the adaptive interface206 may include supervised machine learning, semi-supervised machinelearning, unsupervised machine learning, federated machine learning,collaborative machine learning, and so forth. The supervised machinelearning in the adaptive interface 206 is based on a training datasetwith labeled data, already installed in the adaptive interface 206and/or sent from the API 404 and/or network 110. For the supervisedmachine learning in the adaptive interface 206, the data are labeled andthe algorithms learn to predict the output from the input data, namelyuser data 124.

The semi-supervised learning in the adaptive interface 206 uses a largeamount of user data 124, data of the user device 104, and/or API 404,and only some of preinstalled data and/or data from network 110 by usinga mixture of supervised and unsupervised machine learning techniques.

For unsupervised machine learning in the adaptive interface 122, alldata are unlabeled and the algorithms learn to inherit structure fromthe user data 124, data of the user device 104, and/or API 404.

For the federated machine learning and collaborative machine learning,the adaptive interface 206 collaboratively learns a shared predictionmodel while keeping all the training data on the user device 104 andsensor 106, decoupling the ability to do machine learning from the needto store the data in the network 110. This goes beyond the use of localmodels that make predictions on computing devices by bringing modeltraining to the computing device as well. The user device 104 downloadsthe current model, improves it via adaptive interface 206 by learningfrom user data 124 related to the interaction of the user 102 with theuser device 104 and user data 124 received from the sensor 106, and thensummarizes the changes as a small focused update. Only this update tothe model is sent to the cloud, using encrypted communication, where itis immediately averaged with other user updates to improve the sharedmodel. All the training data remain on the user device 104 and adaptiveinterface 206, and no individual updates are stored in the cloud. Forthe federated and collaborative machine learning setting, the data isdistributed across millions of devices in a highly uneven fashion. Inaddition, these devices have significantly higher-latency andlower-throughput connections and are only intermittently available fortraining.

The user data from the sensor 106 related to biological parameters ofthe user 102 and user interaction with the user device 104 can becommunicated to the network 110 in all machine learning models, but canalso remain as personalized and customized data sets in the adaptiveinterface 206 and user device 104.

The adaptive interface 206 learns about the specifications for routines,data structures, object classes, variables and programming of APIs, andcomputing resource 204, network 110, user device 104, and sensor 106.The adaptive interface 206 processes data about the user interactionwith an application running on the user device 104, such as graphics andaudio from the user device 104, and the user data 124, such asbiological data, from the sensor 106. The adaptive interface 206 learnshow the biometric data and activity data of the user 102 are changing inreal-time when the user 102 interacts with the user device 104. Theadaptive interface 206 dynamically customizes the output data from API404 using the machine learning techniques 126, and sends the customizedoutput data back to the API 404. The API 404 sends the adapted andcustomized output data in real-time to the user device 104. The adaptiveinterface 206 may also collect data associated with the user 102 overtime by receiving the biological data of the user 102 and/or data oninteraction of the user 102 with the user device 104 and analyzes thecollected data using deep learning techniques or other trained learningtechniques. The results of the analysis of the adaptive interface 206and the biological data of the user 102 can be processed and applied tothe output data of the API 404 and the usage of the output data of theAPI 404 in the user device 104 to customize the graphics and audioprovided on the user device 104 to the user 102.

The adaptive interface 206 can customize the output data based on thebiological data of the user 102 (for example, for faster perception timeof the graphics and audio in the user device 104 by the user 102). Theadaptive interface 206 can also customize the graphics and audio to thephysiological state of the user 102 detected and analyzed based on thebiological data of the user 102. For example, a heart rate may be sensedby the sensor 106, and the adaptive interface 206 may customize thegraphics and audio on the user device 104 to decrease or increase theheart rate of the user 102 in real-time while the uses interacts withthe user device 104.

If the sensor 106 is a breathing sensor, the adaptive interface 206 mayemploy thermal imaging and machine learning techniques to adapt andcustomize the graphics and audio to elicit slower or longer inhales andexhales.

The adaptive interface 206 may also send the data of the machinelearning to the network 110. The network 110 can send the data to theserver 108 to update the data of the application running on the userdevice 104. The server 108 can update and improve a code associated withthe application with data associated with the plurality of individualusers. The server 108 can then send the updated output data to theadaptive interface 106 via the network 110.

The adaptive interface 206 can also interact with code storing networkssuch as blockchain 402 or cloud computing (not shown) as alternateimplementations of the server 108 shown on FIG. 1. The adaptiveinterface 206 may send the data of the machine learning to theblockchain 402 for further processing.

In addition to the benefits of APIs, such as scalability, heterogeneousinteroperability, independent evolution of client and server, andempowered clients, the adaptive interface 206 may also add customizationand adaption of the output data in the user device 104 via the user data124 of the user 102. The customization and adaption of the adaptiveinterface 206 can enable faster processing speeds for user interactionwith the application state (graphics, UX, GUI, and audio) as in the userdevice 104. The adaptive interface 206 can also add stress-reduction andchanges of personal biological data via the adapted and customizedoutput data in the user device 104. The adaptive interface 206 can alsoimprove the network speed between adaptive interface 206, API 404, anduser 102 via customized and adapted information processing. The adaptiveinterface 206 may use different types of machine learning techniques toachieve smarter models, lower latency, and less power consumption topotentially ensure privacy when data of user 102 remain on the userdevice 104 and in adaptive interface 206. This approach has anotherimmediate benefit: in addition to providing an update to the sharedmodel to the network 110 and customized output data, the improved andupdated output data on the user device 104 can also be used immediatelyin real-time to provide user experience personalized based on the waythe user 102 uses the user device 104. The adaptive interface 206 canalso replace the API 404 with the API generated based on the outputdata. The adaptive interface 206 can feature all the programming tasksand steps employed in API and replace the API by connecting sensors,user devices, and networks for customized and improved applications,performances, and experiences.

FIG. 5 illustrates an environment 500 within which systems and methodsfor customizing output based on user data can be implemented, inaccordance with some embodiments. The system 200 may be in communicationwith the blockchain 402 and provide the user data 124 to the blockchain402. The blockchain 402 may be in communication with a developercommunity 502 and may provide results of analysis of the user data 124to the developer community 502. The developer community 502 may use theanalysis of the user data 124 to develop further machine learning modelsfor processing the user data 124 by the blockchain 402.

FIG. 6 is a schematic diagram 600 that illustrates operations performedby components of a machine learning system for customizing output basedon user data, according to an example embodiment. The user device 104may display graphics and play audio to a user 102. The user device 104may include a user interface 602, a processing unit 604, an NPU 606, anda displaying unit 608. An adaptive interface 206 of the machine learningsystem 200 for customizing output based on user data may be locatedremotely with respect to the user device 104 (e.g., in a computingcloud). In an example embodiment, the NPU 606 may be not the componentof the user device 104, but may be located remotely with respect to theuser device 104 (e.g., in the computing cloud).

The user 102 may perceive the output in a form of visual data and/oraudio data provided by the user device 104 via the user interface 602,UX, CGI, common gateway interface, a work environment (e.g., a socialmedia platform), and other forms of graphics and potential audio filesor audio-perceived frequencies. In an example embodiment, a screen ofthe user device 104 may display visually-perceived stimuli, forinstance, when the user 104 device uses a retinal implant technology.The user device 104 can be configured in a form of a computing andprocessing unit and may be further configured to provide visual stimuliand play audio-perceived stimuli.

The user 102 may interact with the user device 104. The interaction canbe in any manner, for example, by perceiving, visually or audibly, anapplication running on the user device 104, by changing the visuals oraudio on the user device 104, for example, by haptically interactingwith the user device 104. The user 102 can interact with the user device104 by pressing buttons displayed on the user interface 602 of the userdevice 104 with fingers. In a further example embodiment, pupilmovements of the user 102 or other forms of interaction of the user 102may be tracked.

A sensor 106 may be affixed to the user 102 or may be located inproximity to the user 102 and may sense data related to physicalparameters of the user 102 and convert the data into an electricalsignal. The sensed data may be considered to be an input from the user102. The input may include light, heat, motion, moisture, pressure, orany other physical parameters of the body of the user 102 that can besensed by the sensor 106. The sensor 106 that detects changes of thephysical parameters of the user 102 may be a biosensor configured todetect the presence or concentration of a biological analyte, such as abiomolecule, a biological structure, or a microorganism in/at the bodyof the user 102. The sensor 106 in a form of the biosensor may includethree parts: a component that recognizes the analyte and produces asignal, a signal transducer, and a reader device. The sensor 106 mayprovide an output in a form of a signal that may be transmittedelectronically to the adaptive interface 206 for reading and furtherprocessing.

The sensor 106 may further be a camera configured to detect changes ofphysical parameters of the user 102, such as the color of the skin ofthe user 102. The images can be analyzed using machine learningalgorithms, e.g., using the NPU 606 or the adaptive interface 206, toevaluate the changes of biological parameters of the user 102 (forexample, a heart rate of the user 102). Some types of the sensor 106 mayrequire the use of learning algorithms and machine learning techniquesin the adaptive interface 206, the NPU 606, and/or processing unit 604.The sensor 106 may also be configured on a form of a thermal imagingcamera to detect a stress level, a breathing rate, a heart rate, a bloodoxygen level, and other biological parameters of the user 102. Theadaptive interface 206 may use machine learning algorithms and neuralnetworks to analyze thermal imagery and detect the stress level, thebreathing rate, the heart rate, the blood oxygen level, and otherparameters of the user 102.

The user device 104 may communicate with the sensor 106 to obtain thetime of interaction of the user 102 with the user device 104. Asmentioned above, some types of sensor 106 may use the processing unit604 in the user device 104 for applying the machine learning techniquesand performing the analysis. The biological parameters, the time of thedetection of biological parameters of the user by the sensor 106, anddata related to the user interaction with the user device 104 may besent by the sensor 106 and the user device 104 to the adaptive interface206.

The adaptive interface 206 may customize the data displayed by the userinterface 602 based on data received from the sensor 106 and/or the userdevice 104. The adaptive interface 206 may use the processing units ofthe user device 104, such as a CPU, a GPU, and/or the NPU 606 of theuser device 104. The adaptive interface 206 may use different types ofmachine learning techniques, such as supervised machine learning,semi-supervised machine learning, unsupervised machine learning,federated and/or collaborative machine learning, to customize the outputdata of the user device 104, such as graphics and audio, for the user102 based on the biological data received from the sensor 106 and dataon the user interaction from the user device 104.

The adaptive interface 206 may send the adapted and customized outputdata to the user interface 602. The user interface 602 may display theadapted and customized data on a screen of the user device 104.

The user device 104 may use the displaying unit 608 to display theoutput data in a customized format provided by the adaptive interface206. The cycle of customizing of the output data of the user device 104repeats in real-time so the continuously collected user data and data onuser interaction with the user device 104 are used to update thegraphics and audio of the user device by the adaptive interface 206. Theanalysis of the user data by using machine learning of the adaptiveinterface 206 to adapt the output data in order to elicit the change ofthe biological parameters of the user 102 may result in a fasterprocessing of the user data and customized user experience for the user102 using the user device 140.

As improvements in per-transistor speed and energy efficiency diminish,radical departures from conventional approaches are needed to continueimprovements in the performance and energy efficiency of general-purposeprocessors. One such departure is approximate computing, where an errorin computation is acceptable and the traditional robust digitalabstraction of near-perfect accuracy is relaxed. Conventional techniquesin energy-efficient computing navigate a design space defined by the twodimensions—performance and energy—and traditionally trade one for theother. General-purpose approximate computing explores a thirddimension—error—and trades the accuracy of computation for gains in bothenergy and performance. Techniques to harvest large savings from smallerrors have proven elusive. The present disclosure describes an approachthat uses machine learning-based transformations to accelerateapproximation-tolerant programs. The core idea is to train a learningmodel how an approximable region of a code—a code that can produceimprecise but acceptable results—behaves and replace the original coderegion with an efficient computation of the learned model. Neuralnetworks are used to learn code behavior and approximate the codebehavior. The Parrot algorithmic transformation may be used, whichleverages a simple programmer annotation (“approximable”) to transform acode region from a von Neumann model to a neural model. After thelearning phase, the compiler replaces the original code with aninvocation of a low-power accelerator called an NPU. The NPU is tightlycoupled to the processor to permit profitable acceleration even whensmall regions of code are transformed. Offloading approximable coderegions to the NPU is faster and more energy efficient than executingthe original code. For a set of diverse applications, NPU accelerationprovides whole-application speed increase up to 2.3 times and energysavings of up to 3 times on average with an average quality loss of 9.6%at most. The NPU forms a new class of accelerators and shows thatsignificant gains in both performance and efficiency are achievable whenthe traditional abstraction of near-perfect accuracy is relaxed ingeneral-purpose computing. It is widely understood that energyefficiency now fundamentally limits microprocessor performance gains.

FIG. 7 is a schematic diagram 700 that illustrates operations performedby an adaptive interface to customize output on a user device based onuser data, according to an example embodiment. The user 102 may interactwith a user device 104 by reviewing graphics 702 shown on a userinterface 716 of the user device 104 and listening to audio 704 producedby the user device 104. A sensor 106 may continuously sense user data124 during the interaction of the user 102 with the user device 104.

The user device 104 may provide data related to output data currentlyshown to the user on the user interface 716 to the adaptive interface206. Furthermore, the sensor 106 may provide the sensed user data 124 tothe user device 104, and the user device 104 may analyze the sensed userdata 124 and provide the results of the analysis to the adaptiveinterface 206.

The adaptive interface 206 may process the data related to output datacurrently shown to the user on the user interface 716 and the analyzeduser data 124 using machine learning techniques 126. Based on theprocessing, the adaptive interface 206 may continuously customize outputdata of the user device 104 and provide customized output data 706 tothe API 404 of the user device 104. Upon receipt of the customizedoutput data 706, the API 404 may provide the customized output data 706to the user 102 on a display of the user device 104.

The customized output data 706 may include one or more of the following:an adapted microservice 708, adapted image files 710 (e.g., in JPEG,JPG, GIF, PNG, and other formats), adapted audio files 712 (e.g., inWAV, MP3, WMA, and other formats), adapted files 714 (e.g., in HTM,HTML, JSP, AXPX, PHPH, XML, CSHTML, JS, and other formats), and soforth.

FIG. 8 is a flow chart 800 illustrating customizing output of a userdevice based on user data, in accordance with some example embodiments.At step 802, a user device may receive programming data from an APIassociated with the user device. The user device may provide thereceived programming data to the user at step 804 by displaying graphicson a display of the user device and playing audio using a speaker of theuser device. The user may interact with the user device, e.g., byviewing the graphics displayed on the user device and listening to theaudio played the user device, as shown by step 806. User data, such asbiological data of the user, may be continuously sensed by a sensor asshown by step 808. Additionally, the user device may communicate withthe sensor at step 810 to obtain the time of the user interaction withthe user device. The time of interaction may be used for determining adependency of the user data on the interaction of the user with the userdevice at each moment of time.

At step 812, the user device and the sensor may send user data and dataon user interaction to an adaptive interface. The adaptive interface maycustomize output data of the API of the user device by using machinelearning techniques at step 814. Specifically, the adaptive interfacemay analyze the user data and select changes to be done to the outputdata of the API to cause changing of the user data. For example, theadaptive interface may analyze blood pressure of the user, determinethat the user pressure exceeds a predetermined value, review historicaldata related to dependency of the user pressure on visual and audio dataprovided to the user on the user device, and customize the visual andaudio data to elicit decreasing of the blood pressure of the user.

The adaptive interface may send the customized output data to the API ofthe user device at step 816. Additionally, the adaptive interface maystore the customized output data to a database as historical data. Thehistorical data may be used in further customization of the output data.At step 818, the API may provide the customized output data to the userdevice. The user device may display the customized output data using adisplaying unit at step 820.

FIG. 9 is a schematic diagram 900 showing customization of output dataon a user device based on biological data of a user, according to anexample embodiment. The breath of a user 102 may be continuouslymonitored by a biosensor 902, such as a breath sensor. User datacollected by the biosensor 902 may be provided to a user device 104 asan input 904 from the biosensor 902. The user device 104 may provide theinput 904 from the biosensor 902 to a computing resource. The computingresource may analyze the input 904 from the biosensor 902. In an exampleembodiment, the analysis may include determining breath depth 906 andbreath frequency 908. The computing resource may provide the results ofthe analysis to an adaptive interface. Additionally, the user device 104may also provide, to an adaptive interface, data related to output 910viewed by the user 102 at the time the biosensor 902 collected the userdata. The output 910 provided to the user 102 on the user device 104 mayinclude graphics 912 shown to the user 102 using the user device 104,such as background, fonts, GUI elements, CGI, UX, and the like.

The adaptive interface may process the results of the analysis, theoutput 910 viewed by the user 102, and historical data previouslycollected for the user 102 and/or a plurality of users on dependency ofbiological data of the user 102 and/or the plurality of users on theoutput 910 viewed on the user device 104. The adaptive interface mayperform the processing 922 using machine learning techniques. Based onthe processing, the adaptive interface may customize the output 910 toprovoke changing of the user data (e.g., to provoke deeper inhales ofthe user 102). The adaptive interface may provide customized output 914to the user device 104. The customized output 914 provided to the user102 on the user device 104 may include customized graphics 916 shown tothe user 102 using the user device 104, such as customized background,fonts, GUI elements, CGI, UX, and the like.

The biosensor 902 may continue monitoring the user data and provide thedata collected based on the monitoring to the computing resource. Thecomputing resource may analyze the input 904 from the biosensor 902. Inan example embodiment, the analysis may include determining breath depth918 and breath frequency 920 that the user has after reviewing thecustomized output 914.

The user data continuously collected by the biosensor 902 and thecustomized output 914 provided to the user 102 may be continuouslyanalyzed by the adaptive interface. The adaptive interface may continueapplying the machine learning techniques for the analysis 924. Based onthe analysis 924 and machine learning processing 922 in real-time, theelements of graphics 928 provided to the user 102 on a user interface ofthe user device 104 may be continuously modified, as shown by block 926,to elicit improved breathing of the user 102. The breath depth 930 andbreath frequency 932 of the user 102 may be continuously analyzed by theadaptive interface for further modification of the output of the userdevice 102.

FIG. 10 is a schematic diagram 1000 illustrating processing data from asensor using machine learning techniques, according to an exampleembodiment. The sensor, such as a breath sensor, may provide detecteddata as an input to an adaptive interface. The adaptive interface mayprocess the input from the sensor in a machine learning environmentusing machine learning algorithms. The adaptive interface maycontinuously learn about user response to providing customized visualand audio output data to the user using the user device. The userresponse may include changing of the biological parameters of the userinvoked by reviewing the customized visual and audio output data by theuser. The biological parameters may include an average breath force 1004sensed by the sensor and an average breath force 1006 and 1008 furthersensed by the sensor upon providing the customized visual and audiooutput data to the user.

FIG. 11 is a flow chart 1100 illustrating continuous customization ofoutput based on user data, according to an example embodiment. Datarelated to biological parameters of the user and interaction of the userwith a user device may be continuously captured by a sensor and the userdevice in a form of user biodata and interaction 1102. The capturedbiological data, also referred to as biodata, and data on interaction1102 may be provided to an adaptive interface 206 as input 1104. At step1106, the adaptive interface 206 may detect the user biodata andinteraction 1102 and send the data to be displayed by a user device asoutput 1108. A displaying unit of the user device may process the datareceived from the adaptive interface 206 and provide the output on theuser device, as shown by block 1110. The output may be provided bydisplaying visual data, playing audio data, providing haptic feedback,and so forth.

The sensor and the user device may continuously provide further userbiodata and data on interaction, as shown by blocks 1112, 1114, 1116,and 1118. The adaptive interface 206 may apply the machine learningtechniques to customize the output of the user device and send thecustomized output to the user device, as shown by blocks 1120, 1122,1124, and 1126. The displaying unit of the user device may process thecustomized data received from the adaptive interface 206 and provide theupdated data on the user device, as shown by blocks 1128, 1130, 1132,1134, and 1136.

FIG. 12 is a schematic diagram 1200 showing operations performed by anadaptive interface to continuously customize output data using machinelearning techniques, according to an example embodiment. The adaptiveinterface may continuously receive input 1202 from a user device. Theinput 1202 may include user data sensed by a sensor and time ofproviding data, e.g., graphics, on a display of the user device to theuser. At block 1204, based on the input 1202, the adaptive interface maydetermine which user data the user had at a time of providing the dataon the display. For example, the adaptive interface may determine theblood pressure the user had when the user read information on a greenfont of a webpage displayed on the user device. The adaptive interfacemay apply machine learning techniques and neural networks 1206 todetermine whether the user data need to be changed according topredetermined criteria (e.g., whether the blood pressure of the user isabove a predetermined value at the current moment of time). The adaptiveinterface may further apply machine learning techniques and neuralnetworks 1206 to determine specific changes 1208 to be applied to theoutput data of the user device to cause changing of the user data (e.g.,to cause decreasing of the blood pressure of the user). The adaptiveinterface may send the changed data to be displayed to the user device.A displaying unit of the user device may process and display the changeddata as the output 1210 of the user device.

The adaptive interface may continue receiving input 1202 from the userdevice. Specifically, the adaptive interface may receive user datasensed by the sensor and time of providing changed data on the displayof the user device to the user. Based on the input 1202, the adaptiveinterface may determine which user data the user had at a time ofproviding the changed data on the display, as shown by block 1212. Theadaptive interface may determine whether the user data still needs to bechanged (e.g., if the blood pressure of the user is still above thepredetermined value). If the user data still needs to be changed, theadaptive interface may determine, at block 1214, which adjustments ofdata to be displayed to the user need to be made. The adaptive interfacemay send the adjusted data to be displayed to the user device. Thedisplaying unit of the user device may process and display the adjusteddata as the output 1210 of the user device.

Upon providing the adjusted data, the adaptive interface may continuereceiving input 1202 from the user device. Specifically, the adaptiveinterface may receive user data sensed by the sensor and time ofproviding the adjusted data on the display of the user device to theuser. Based on the input 1202, the adaptive interface may determinewhich user data the user had at a time of providing the adjusted data onthe display, as shown by block 1216. At block 1218, the adaptiveinterface may determine whether the adjustment of data to be displayedto the user led to a personalized change (i.e. to the change ofbiological parameters of the user). The adaptive interface may performcontinuous adjustment of data to be displayed to the user, as shown byblock 1220. The adaptive interface may continuously provide the adjusteddata as the output 1210 to the user device.

FIG. 13 is a schematic diagram 1300 showing operations performed by anadaptive interface to continuously customize output data using machinelearning techniques, according to an example embodiment. The adaptiveinterface may continuously receive input 1302 from a user device. Theinput 1302 may include user data sensed by a sensor and time ofproviding data, e.g., output data in a form of graphics, on a display ofthe user device to the user. Based on the input 1302, the adaptiveinterface may determine which user data the user had at a time ofproviding the data on the display at block 1304. The adaptive interfacemay apply machine learning techniques and neural networks 1306 todetermine whether the user data needs to be changed according topredetermined criteria. The adaptive interface may further apply machinelearning techniques and neural networks 1306 to determine specificchanges 1308 to be applied to the output data, e.g., graphics, audio, orolfactory data, of the user device to cause changing of the user data.The adaptive interface may send the changed data to be displayed in afrontend, i.e., on a display of the user device. A displaying unit ofthe user device may process and display the changed data as the output1310 of the user device.

The adaptive interface may receive further input 1312 from the userdevice. The input 1312 may include changed user data sensed by thesensor and time of providing changed data on the display of the userdevice to the user. Based on the input 1312, the adaptive interface maydetermine whether the change of the output data led to personalized ordesired change of the user data, as shown by block 1314. Specifically,the adaptive interface may determine in which way the user data changedin response to providing the changed output data to the user. Theadaptive interface may further apply machine learning techniques andneural networks 1316 to determine adjustments 1318 to be applied to thechanged output data of the user device to cause further changing of theuser data. The adaptive interface may send the adjusted data to bedisplayed in the frontend (i.e., on the display of the user device). Thedisplaying unit of the user device may process and display the adjusteddata as the output 1310 of the user device.

FIG. 14 is a block diagram 1400 illustrating continuous personalizationof a brightness level on a user device based on data related torespiration or heart rate of a user, according to an example embodiment.The respiration may be determined based on respiratory muscle movements,thermal changes of skin, movement of belly or chest, a heart rate, andso forth. An input 1402 may be continuously provided to an adaptiveinterface 206. The adaptive interface 206 may process the input 1402 andprovide an output 1404 for displaying on the user device.

Specifically, user data 1406 may be provided as the input 1402 to theadaptive interface 206. The user data 1406 may include data related torespiration or a heart rate of the user at the time of interaction ofthe user with the user device having a particular brightness level. Theuser data 1406 may further include a time when the user interacted withor perceived the particular brightness level of the user device. Theadaptive interface 206 may determine, at block 1408, which user data theuser had at a particular time when the user device had a particularbrightness level (e.g., what respiration and heart rate the user had atthe time when the brightness level of the user device was 5) as shown byblock 1410.

At block 1412, the adaptive interface 206 may change the brightnesslevel on a scale from 1 to 10 to cause the change of the respiration ofthe heart rate of the user. The determination whether the brightnesslevel needs to be changed and to what value may be made using machinelearning techniques based on historical data of the user or a pluralityof users. Upon setting the brightness level 1414 of the user device from1 to 10, the adaptive interface 206 may receive further user input data1416. The further user data 1416 may include data related to respirationor the heart rate of the user at the time of interaction of the userwith the user device having the brightness level 1414 from 1 to 10. Theuser data 1406 may further include a time when the user interacted withor perceived the brightness level 1414 from 1 to 10 of the user device.

The adaptive interface 206 may determine, at block 1418, which user datathe user had at particular time when the user device had the brightnesslevel 1414 (e.g., what respiration and heart rate the user had at thetime when the brightness level 1414 of the user device was from 1 to10). At block 1420, the adaptive interface 206 may select an adjusted,i.e., personalized, value of brightness level intended, for example, toslow the respiration or the heart rate of the user. The personalizedbrightness level 1422 (e.g., 3-4) selected by the adaptive interface 206may be set on the user device.

Upon setting the personalized brightness level 1422 of the user device,the adaptive interface 206 may receive continuously detected user data1424. At block 1426, the adaptive interface 206 may determine which userdata the user had at a particular time when the user device had thepersonalized brightness level 1422. The adaptive interface 206 mayperform continuous personalization of the brightness level at block 1428to elicit a personalized change of user data, such as the respiration orthe heart rate of the user.

FIG. 15 is a block diagram 1500 illustrating continuous personalizationof a volume level on a user device based on data related to respirationor a heart rate of a user, according to an example embodiment. An input1502 may be continuously provided to an adaptive interface 206. Theadaptive interface 206 may process the input 1502 and provide an output1504 for displaying on the user device.

Specifically, user data 1506 may be provided to the adaptive interface206. The user data 1506 may include data related to respiration or theheart rate of the user at the time of interaction of the user with theuser device having a particular volume level. The user data 1506 mayfurther include a time when the user interacted with or perceived theparticular volume level of the user device. The adaptive interface 206may determine, at block 1508, which user data the user had at aparticular time when the user device had a particular volume level(e.g., what respiration and heart rate the user had at the time when thevolume level of the user device was 5), as shown by block 1510.

At block 1512, the adaptive interface 206 may change the volume level ona scale from 1 to 10 to cause the change of the respiration of heartrate of the user. The determination whether the volume level needs to bechanged and to what value may be made using machine learning techniquesbased on historical data of the user or a plurality of users. Uponsetting the volume level 1514 of the user device from 1 to 10, theadaptive interface 206 may receive further user input data 1516. Thefurther user data 1516 may include data related to respiration or theheart rate of the user at the time of interaction of the user with theuser device having the volume level 1514 from 1 to 10. The user data1506 may further include time when the user interacted with or perceivedthe volume level 1514 from 1 to 10 of the user device.

The adaptive interface 206 may determine, at block 1518, which user datathe user had at a particular time when the user device had the volumelevel 1514 (e.g., what respiration and heart rate the user had at thetime when the volume level 1514 of the user device was from 1 to 10). Atblock 1520, the adaptive interface 206 may select an adjusted value ofthe volume level intended, for example, to slower the respiration or theheart rate of the user. The personalized volume level 1522 (e.g., 3-4)selected by the adaptive interface 206 may be set on the user device.

Upon setting the personalized volume level 1522 of the user device, theadaptive interface 206 may receive continuously detected user data 1524.At block 1526, the adaptive interface 206 may determine which user datathe user had at particular time when the user device had thepersonalized volume level 1522. The adaptive interface 206 may performcontinuous personalization of the volume level at block 1528 to elicit apersonalized change of user data, such as the respiration or the heartrate of the user.

FIG. 16 is a block diagram 1600 illustrating continuous personalizationof an odorant level on a user device based on data related torespiration or a heart rate of a user, according to an exampleembodiment. An input 1602 may be continuously provided to an adaptiveinterface 206. The adaptive interface 206 may process the input 1602 andprovide an output 1604 for displaying on the user device. The userdevice may include an artificial olfaction device 122 as shown on FIG.1.

User data 1606 may be provided to the adaptive interface 206. The userdata 1606 may include data related to respiration or the heart rate ofthe user at the time of interaction of the user with the user devicehaving a particular volume level. The user data 1606 may further includea time when the user interacted with or perceived the particular odorantlevel of the user device. The adaptive interface 206 may determine, atblock 1608, which user data the user had at a particular time when theuser device had a particular odorant level (e.g., what respiration andheart rate the user had at the time when the odorant level of the userdevice was 5), as shown by block 1610.

At block 1612, the adaptive interface 206 may change the odorant levelon a scale from 1 to 10 to cause the change of the respiration of heartrate of the user. The determination whether the odorant level needs tobe changed and to what value may be made using machine learningtechniques based on historical data of the user or a plurality of users.Upon setting the odorant level 1614 of the user device from 1 to 10, theadaptive interface 206 may receive further user input data 1616. Thefurther user data 1616 may include data related to respiration or theheart rate of the user at the time of interaction of the user with theuser device having the odorant level 1614 from 1 to 10. The user data1616 may further include a time when the user interacted with orperceived the odorant level 1614 from 1 to 10 of the user device.

The adaptive interface 206 may determine, at block 1618, which user datathe user had at a particular time when the user device had the odorantlevel 1614 (e.g., what respiration and heart rate the user had at thetime when the odorant level 1614 of the user device was from 1 to 10).At block 1620, the adaptive interface 206 may select a personalizedvalue of the odorant level intended, for example, to slow therespiration or the heart rate of the user. The personalized odorantlevel 1622 (e.g., 3-4) selected by the adaptive interface 206 may be seton the user device.

Upon setting the personalized odorant level 1622 of the user device, theadaptive interface 206 may receive continuously detected user data 1624.At block 1626, the adaptive interface 206 may determine which user datathe user had at a particular time when the user device had thepersonalized odorant level 1622. The adaptive interface 206 may performcontinuous personalization of the odorant level at block 1628 to elicita personalized change of user data, such as the respiration or the heartrate of the user.

FIG. 17 is a schematic diagram 1700 showing a user interface of a mobiledevice customized by a machine learning system for customizing outputbased on user data, according to an example embodiment. Specifically,FIG. 17 illustrates customizing a graphics output on a user device 104based on user data 124 sensed by a sensor 106. A user interface 1702 maydisplay output data 1704 on a screen of the user device 104. Theadaptive interface 206 of the machine learning system 200 forcustomizing output based on user data may customize the output data 1704and send customized output data 1706 to the user interface 1702. Theuser interface 1702 may display the customized data 1706 on the screenof the user device 104. The customized output data 1706 may include achanged font, changed colors, changed brightness, changed contrast, andthe like.

Upon further customization of the output data, the adaptive interface206 may send further customized output data 1708 to the user interface1702. The user interface 1702 may display the further customized data1708 on the screen of the user device 104. The customized output data1708 may include a changed font, changed colors, changed brightness, achanged contrast, a changed background, and the like.

The adaptive interface 206 may continuously customize the output dataand provide the further customized output data 1710 to the userinterface 1702. The user interface 1702 may display the furthercustomized data 1710 on the screen of the user device 104.

FIG. 18 is a schematic diagram 1800 showing output data of a user devicecustomized by a machine learning system for customizing output based onuser data, according to an example embodiment. Specifically, FIG. 18illustrates customizing an audio output on headphones 118 based on userdata 124 of a user 102 sensed by a sensor 106. The output data, such asa pitch 1802 and a volume 1804 of the sound, may be provided to theheadphones 118. The adaptive interface 206 of the machine learningsystem 200 for customizing output based on user data may customize thepitch 1802 and the volume 1804 and send data associated with customizedpitch 1806 and customized volume 1808 to the headphones 118. Theheadphones 118 may reproduce the audio output with the customized pitch1806 and customized volume 1808.

Upon further customization of the audio output based on the user data124, the adaptive interface 206 may send further customized pitch 1810and further customized volume 1812 to the headphones 118. The headphones118 may reproduce the audio output with the further customized pitch1810 and further customized volume 1812.

The adaptive interface 206 may continuously customize the audio outputand provide the further customized pitch 1814 and further customizedvolume 1816 to the headphones 118. The headphones 118 may reproduce thefurther customized pitch 1814 and further customized volume 1816 to theuser 102.

FIG. 19 is a schematic diagram 1900 showing output data of a user devicecustomized by a machine learning system for customizing output based onuser data, according to an example embodiment. Specifically, FIG. 19illustrates customizing olfactory data on an artificial olfaction device120 based on user data 124 of a user 102 sensed by a sensor 106. Theoutput data, such as units 1902 of a perceptual axis 1904 of odorantpleasantness that ranges from very pleasant (e.g., rose as shown byelement 1906) to very unpleasant (e.g., skunk as shown by element 1908),may be provided to the user 102 by the artificial olfaction device 120.The adaptive interface 206 of the machine learning system 200 forcustomizing output based on user data may customize the units 1902 ofthe perceptual axis 1904 and send customized units 1910 to theartificial olfaction device 120. The artificial olfaction device 120 mayset the olfactory data according to the customized units 1910.

Upon further customization of the olfactory data based on the user data124, the adaptive interface 206 may send further customized units 1912of the perceptual axis 1904 to the artificial olfaction device 120. Theartificial olfaction device 120 may set the olfactory data according tothe customized units 1912.

The adaptive interface 206 may continuously customize the olfactory dataand provide the further customized units 1914 of the perceptual axis1904 to the artificial olfaction device 120. The artificial olfactiondevice 120 may set the olfactory data according to the customized units1914.

FIG. 20 is a schematic diagram 2000 showing customizing output of a userdevice based on user data captured by a digital camera of the userdevice, according to an example embodiment. The user device may includea smartphone. The digital camera shown as camera 2002 may be disposed ata distance 2004 from the user 102. The distance 2004 at which the camera2002 may be configured to capture user data may be up to several metersor any other distance depending on parameters of the camera 2002 andenvironmental conditions.

In an example embodiment, the camera 2002 may be selected from acharge-coupled device, a complementary metal-oxide semiconductor imagesensor, or any other type of an image sensor. The camera 2002 of theuser device may be used as a non-contact and non-invasive device tomeasure user data. The user data may include a respiratory rate, pulserate, blood volume pulse, and so forth. The camera 2002 may be used tocapture an image the user 102. The user data captured by the camera 2002may be processed by the adaptive interface 206 of the system 200. Theprocessing may be performed using CPU, GPU, and/or NPU. The user datacaptured by the camera 2002 may be the input for the adaptive interface206 and may be processed together with the data concerning the time ofthe visuals displayed to the user and audio provided to the user at thetime of capture of the user data, and together with the data concerningthe time of the recording of the user data.

In an example embodiment, the respiratory rate, the heart rate, and theblood volume pulse may be sensed simultaneously using the camera 2002.Specifically, the camera 2002 may capture an image 2006 of the user.Upon capturing, a part of a skin 2008 of the user 202 may be detected onthe image 2006. Upon detecting the image of the skin, a region ofinterest 2010 may be selected as shown by step 2012. After the selectionof the region of interest 2012, the changes in the average imagebrightness of the region of interest 2012 for a short time can bemeasured. The selection of the region of interest 2012 of the face maybe used to obtain blood circulation features and obtain the raw bloodvolume pulse signal. The selection of a region of interest may influencethe following heart rate detection steps. First, the selection of aregion of interest may affect the tracking directly since a commonlyapplied tracking method uses a first frame of the captured region ofinterest. Second, the selected regions of interest are regarded as thesource of cardiac information. The pixel values inside the region ofinterest can be used for intensity-based methods, while feature pointlocations inside the region of interest can be used for motion-basedmethods.

The time-lapse image of a part of the skin of the user 102 may beconsecutively captured, and the changes in the average brightness of theregion of interest 2010 can be measured for a period of time, forexample, for 30 seconds. The brightness data can be processed by aseries of operations of interpolation using a first-order derivative, alow pass filter of 2 Hz, and a sixth-order auto-regressive spectralanalysis.

Remote photoplethysmography may be used for contactless monitoring ofthe blood volume pulse using the camera. Blood absorbs light more thanthe surrounding tissues and variations in blood volume affect lighttransmission and reflectance as schematically shown by an arrow 2034.This leads to the subtle color changes in human skin, which areinvisible to human eyes but can be recorded by the camera. Variousoptical models can be applied to extract the intensity of color changescaused by pulse.

It is possible to capture heart rate signals at a frame rate of eightframes per second (fps), under the hypothesis that the human heartbeatfrequency lies between 0.4 and 4 Hz. A frame rate between 15 and 30 fpsis sufficient for heart rate detection. The estimation of the heart rateis performed by directly applying noise reduction algorithms and opticalmodeling methods. Alternatively, the usage of manifold learning methodsmapping multidimensional face video data into one-dimensional space canbe used to reveal the heart rate signal. Hemoglobin and oxyhemoglobinboth have the ability of absorption in the green color range and low inthe red color range. However, all three color channels (red, green, andblue) contain photoplethysmogram (PPG) information. Red green blue (RGB)color filter 2036 can be used to extract red frames 2014, green frames2016, and blue frames 2018 from the captured image of interest 2010. Redsignal 2020, green signal 2022, and blue signal 2024 can be determinedbased on the captured red frames 2014, green frames 2016, and blueframes 2018.

FIG. 21 is a schematic diagram 2100 showing an analysis of captured userdata by an adaptive interface, according to an example embodiment.Intensity-based methods 2106, 2108, 2110 can be used to process PPGsignals captured by the camera. Normalized color intensity (RGB colors)can be analyzed. Using auto-regressive spectral analysis, two clearpeaks can be detected at approximately 0.3 and 1.2 Hz. The peakscorrespond to the respiratory rate and the heart rate. The peak 2102with the frequency of 0.3 Hz corresponds to the respiratory rate, andthe peak 2104 with the frequency of 1.2 Hz corresponds to the heartrate. The green channel provides the strongest signal-to-noise ratio.Consequently, the green channel can be used for extracting the heartrate.

Referring back to FIG. 20, upon detecting the red signal 2020, greensignal 2022, and blue signal 2024, an average RGB signal can bedetermined at step 2026. The signal de-trending of the average RGBsignal can be performed at step 2028. The processed signal can benormalized at step 2030. Filtering of the normalized signal may beperformed at step 2030.

Referring again to FIG. 21, the analysis of captured user data canfurther include capturing brightness level, contrast level, saturationlevel, and vibrance level of data shown to the user on the user device.The user perceives the visuals shown on the user device with acontinuously changing degree of brightness level, contrast level,saturation level, and vibrance level. The adaptive interface may receivean input in a form of the time of the displaying of the degree or levelof brightness, contrast, saturation and vibrance, and the time of theheart rate and respiratory rate as analyzed from user data detected bythe camera. As shown in FIG. 21, the inputs are levels of brightness2112, contrast 2114, saturation 2116, and vibrance 2118 that may bemapped to the time, as well as may be mapped to the time and analysis2120, 2122, 2124 of values of heart rate and respiratory rate.

The adaptive interface can perform a continuous processing 2126 using aneural processing unit for predictive modeling of the datasets capturedby the sensor (the camera) and the visual adjustments. In the adaptiveinterface, the input of datasets may be processed using deep learningtechniques. The deep learning technique may apply specific and differingmachine learning techniques to the datasets to learn how to process thedatasets and adapt the visual adjustments on the user device to supportslower heart rate and respiratory rate. Machine learning techniques canbe supervised, semi-supervised, and unsupervised. The adaptive interfacemay analyze correlations between visual adjustments and heart rate andrespiratory rates, process the datasets, and create predictive models toadapt the visual adjustments to the desired outcome of slower heart rateand slower respiratory rate personalized to the user in real-time andcontinuously. For the inputs, the adaptive interface may identifyfeatures of the visual adjustments that are predictive of the outcomesto predict heart rate and respiratory rate. The adaptive interface mayuse classification, regression, clustering, convolutional neuralnetworks, and other machine learning techniques based on which thedatasets are analyzed and the customized output data are predicted forthe fastest lowering and slowing of the heart rate and respiratory rateof the user. The probability and predictive modeling performed by theadaptive interface may be adaptive to learn how to adapt the visualadjustments of the user device to the user with varying heart rate andrespiratory rate. As the adaptive interface identifies patterns in thedatasets of the visual adjustments and heart rate and respiratory rate,the adaptive interface may learn from the observations. When exposed tomore observations, the predictive performance of the adaptive interfacemay be improved.

In an example embodiment, the analysis performed by the adaptiveinterface may include step 2126, at which signal extraction may beperformed. The signal extraction may include detection, definition, andtracking of a region of interest, raw signal extraction to obtain rawsignal. The visual adjustments may be selected based on the signalextraction. The visual adjustments may include adjustments of thevibrance level, saturation level, contrast level, brightness level.Based on the visual adjustments, the display spectrum may be determinedand applied to a display of the user device.

The analysis may further include step 2128, at which signal estimationmay be performed by applying filtering, dimensionality, and reduction tothe signal to obtain an RGB signal. Furthermore, the heart rate andrespiratory rate may be estimated using frequency analysis and peakdetection.

The analysis may further include step 2130, at which adaptive modelingmay be performed. The adaptive modeling may include deep learningtechniques, machine learning techniques for adaptation of visuals basedon the heart rate and respiratory rate, model learning using regression,clustering, feature selection, and convolutional neural networks.

The analysis may further include step 2132, at which adaptiveimplementation may be performed. Specifically, output data for the userdevice may be provided, model testing may be performed, levels ofvibrance, saturation, contrast, and brightness may be adapted to theheart rate and respiratory rate in real-time and continuously. Machinelearning and other techniques may be continuously applied.

FIG. 22 is a schematic diagram 2200 showing output data continuouslyadapted by an adaptive interface, according to an example embodiment.The adaptive interface 206 may send the analyzed, adapted and customizedoutput data to a CPU or GPU for processing of the personalized datasets.The user 102 may be presented with user interfaces 2202, 2204, 2206,2208, 2210, which may be continuously personalized in real-time. Theuser interfaces 2202, 2204, 2206, 2208, 2210 may have varyingbrightness, contrast, saturation, and vibrance levels. The user device104 may also provide to the adaptive interface data relating to aprocessing speed. Therefore, the adaptive interface may learn how toimprove the data processing for faster processing so that the visualadjustments for slower heart rate and respiratory rate can be processedfaster and more efficiently.

The adaptive interface may receive updates from a network and databaseabout methods to analyze and process the datasets. The updates may bebased on scientific studies and tests as to what visuals are supportiveto slower the heart rate and the respiratory rate. The updates may alsoinclude data on how the heart rate and the respiratory rate can beanalyzed and what a “slower” heart rate and respiratory rate means. Thefocus of the adaptive interface may be to provide a slower, calmer, anddeeper heart rate and respiratory rate. The definition of a slower,calmer, and deeper heart rate and respiratory rate can change over timebased on scientific and other data, and the adaptive interface may beupdated from the network to integrate these changes and adapt theprocessing of the input data to the updates. Each adaptive interfaceused by each of a plurality of users may provide data relating todatasets and processing to a database, so the machine learningtechniques may use data from a plurality of adaptive interfacesassociated with the plurality of users to improve the visual adjustmentsto the heart rate and respiratory rate of the user. The updates relatingto data learned from the plurality of adaptive interfaces and users maybe provided to each adaptive interface.

The adaptive interface may be directed to analyzing and evaluating whichdatasets of visual adjustments, i.e. output data, personalize thevisuals shown to the user on the user device to personalize a desiredchange of heart rate and respiratory rate. The heart rate or respiratoryrate can be customized to be lower than the average and/or firstdetected heart rate and respiratory rate. The adaptive interface maycontinuously analyze the user data and interaction of the user with theuser device and visuals, and may continuously learn to customize thevisuals with the visual adjustments to further change the heart rate andrespiratory rate of the user.

FIG. 23 is a flow chart 2300 showing a method for customizing outputbased on user data, according to an example embodiment. The adaptiveinterface may continuously process user data, as shown by operation2302. At operation 2304, the adaptive interface may determine whetherthe heart rate and the respiratory rate of the user slow down. If theheart rate and the respiratory rate do not slow down, the method maycontinue with operation 2306, at which the adaptive interface may changevisuals, i.e., the output data on the user device, using adjustments ofbrightness level, contrast level, saturation level, and vibrance levelof data shown to the user on the user device.

At operation 2308, data related to the changed visuals may be processedand provided to the user on the user device. At operation 2310, theadaptive interface may again determine whether the heart rate and therespiratory rate of the user slow down. If the heart rate and therespiratory rate slow down, the adaptive interface may perform operation2312, at which data related to the visuals with adjustments ofbrightness level, contrast level, saturation level, and vibrance levelmay be processed. At operation 2314, the adaptive interface may continueprocessing of the user data.

If it is determined at operation 2310 that the heart rate and therespiratory rate do not slow down, the adaptive interface may employother machine learning techniques at operation 2316 to perform furtheradaptation of the visuals.

Returning to operation 2304, if it is determined that the heart rate andthe respiratory rate slow down, the adaptive interface may determine, atoperation 2316, whether tendencies of adjustment of visuals supportslower heart rate and respiratory rate. If the tendencies of adjustmentof visuals support slower heart rate and respiratory rate, the adaptiveinterface may continue with operation 2308. If tendencies of adjustmentof visuals do not support slower heart rate and respiratory rate, theadaptive interface may perform operation 2318, at which the adaptiveinterface may search for adjustments of visuals that may support slowerheart rate and slower respiratory rate. Upon selecting adjustments ofvisuals, the adaptive interface may continue with operation 2312.

FIG. 24 illustrates an exemplary computing system 2400 that may be usedto implement embodiments described herein. The exemplary computingsystem 2400 of FIG. 24 may include one or more processors 2410 andmemory 2420. Memory 2420 may store, in part, instructions and data forexecution by the one or more processors 2410. Memory 2420 can store theexecutable code when the exemplary computing system 2400 is inoperation. The exemplary computing system 2400 of FIG. 24 may furtherinclude a mass storage 2430, portable storage 2440, one or more outputdevices 2450, one or more input devices 2460, a network interface 2470,and one or more peripheral devices 2480.

The components shown in FIG. 24 are depicted as being connected via asingle bus 2490. The components may be connected through one or moredata transport means. The one or more processors 2410 and memory 2420may be connected via a local microprocessor bus, and the mass storage2430, one or more peripheral devices 2480, portable storage 2440, andnetwork interface 2470 may be connected via one or more input/outputbuses.

Mass storage 2430, which may be implemented with a magnetic disk driveor an optical disk drive, is a non-volatile storage device for storingdata and instructions for use by a magnetic disk or an optical diskdrive, which in turn may be used by one or more processors 2410. Massstorage 2430 can store the system software for implementing embodimentsdescribed herein for purposes of loading that software into memory 2420.

Portable storage 2440 may operate in conjunction with a portablenon-volatile storage medium, such as a compact disk (CD) or digitalvideo disc (DVD), to input and output data and code to and from thecomputing system 2400 of FIG. 24. The system software for implementingembodiments described herein may be stored on such a portable medium andinput to the computing system 2400 via the portable storage 2440.

One or more input devices 2460 provide a portion of a user interface.The one or more input devices 2460 may include an alphanumeric keypad,such as a keyboard, for inputting alphanumeric and other information, ora pointing device, such as a mouse, a trackball, a stylus, or cursordirection keys. Additionally, the computing system 2400 as shown in FIG.24 includes one or more output devices 2450. Suitable one or more outputdevices 2450 include speakers, printers, network interfaces, andmonitors.

Network interface 2470 can be utilized to communicate with externaldevices, external computing devices, servers, and networked systems viaone or more communications networks such as one or more wired, wireless,or optical networks including, for example, the Internet, intranet, LAN,WAN, cellular phone networks (e.g., Global System for Mobilecommunications network, packet switching communications network, circuitswitching communications network), Bluetooth radio, and an IEEE802.11-based radio frequency network, among others. Network interface2470 may be a network interface card, such as an Ethernet card, opticaltransceiver, radio frequency transceiver, or any other type of devicethat can send and receive information. Other examples of such networkinterfaces may include Bluetooth®, 3G, 4G, and WiFi® radios in mobilecomputing devices as well as a USB.

One or more peripheral devices 2480 may include any type of computersupport device to add additional functionality to the computing system.The one or more peripheral devices 2480 may include a modem or a router.

The components contained in the exemplary computing system 2400 of FIG.24 are those typically found in computing systems that may be suitablefor use with embodiments described herein and are intended to representa broad category of such computer components that are well known in theart. Thus, the exemplary computing system 2400 of FIG. 24 can be apersonal computer, hand held computing device, telephone, mobilecomputing device, workstation, server, minicomputer, mainframe computer,or any other computing device. The computer can also include differentbus configurations, networked platforms, multi-processor platforms, andso forth. Various operating systems (OS) can be used including UNIX,Linux, Windows, Macintosh OS, Palm OS, and other suitable operatingsystems.

Some of the above-described functions may be composed of instructionsthat are stored on storage media (e.g., computer-readable medium). Theinstructions may be retrieved and executed by the processor. Someexamples of storage media are memory devices, tapes, disks, and thelike. The instructions are operational when executed by the processor todirect the processor to operate in accord with the example embodiments.Those skilled in the art are familiar with instructions, processor(s),and storage media.

It is noteworthy that any hardware platform suitable for performing theprocessing described herein is suitable for use with the exampleembodiments. The terms “computer-readable storage medium” and“computer-readable storage media” as used herein refer to any medium ormedia that participate in providing instructions to a CPU for execution.Such media can take many forms, including, but not limited to,non-volatile media, volatile media, and transmission media. Non-volatilemedia include, for example, optical or magnetic disks, such as a fixeddisk. Volatile media include dynamic memory, such as RAM. Transmissionmedia include coaxial cables, copper wire, and fiber optics, amongothers, including the wires that include one embodiment of a bus.Transmission media can also take the form of acoustic or light waves,such as those generated during radio frequency and infrared datacommunications. Common forms of computer-readable media include, forexample, a floppy disk, a flexible disk, a hard disk, magnetic tape, anyother magnetic medium, a CD-read-only memory (ROM) disk, DVD, any otheroptical medium, any other physical medium with patterns of marks orholes, a RAM, a PROM, an EPROM, an EEPROM, a FLASHEPROM, any othermemory chip or cartridge, a carrier wave, or any other medium from whicha computer can read.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to a CPU for execution. Abus carries the data to system RAM, from which a CPU retrieves andexecutes the instructions. The instructions received by system RAM canoptionally be stored on a fixed disk either before or after execution bya CPU.

Thus, machine learning systems and methods for customizing output basedon user data are described. Although embodiments have been describedwith reference to specific exemplary embodiments, it will be evidentthat various modifications and changes can be made to these exemplaryembodiments without departing from the broader spirit and scope of thepresent application. Accordingly, the specification and drawings are tobe regarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A machine learning system for customizing outputbased on user data, the system comprising: at least one sensorconfigured to continuously capture the user data associated with a userduring perception of output data by the user; at least one computingresource comprising a first processor and a first memory, the at leastone computing resource being configured to: analyze the user datareceived from the at least one sensor; and based on the analysis,determine dependencies between the user data and the output data; and anadaptive interface comprising a second processor and a second memory,the adaptive interface being configured to continuously customize theoutput data using at least one machine learning technique based on theanalysis of the user data and the dependencies, the customized outputdata intended to elicit an automatic and immediate personalized changeof a biological response of the user to perception of the customizedoutput data by the user without requiring the user to take an action inresponse to the customized output data.
 2. The system of claim 1,wherein the user data includes at least one of the following: biologicaldata of a user, biological data of a plurality of users, historical dataof the user, historical data of the plurality of users, and ambientdata.
 3. The system of claim 2, wherein the automatic and immediatepersonalized change includes a change in the biological data.
 4. Thesystem of claim 2, wherein the biological data includes at least one ofthe following: a respiratory rate, a heart rate, a heart ratevariability, an electroencephalography, an electrocardiography, anelectromyography, an electrodermal activity, a mechanomyography, ahaptic interaction, a motion, a gesture, pupil movement, a biologicalanalyte, a biological structure, a microorganism, a color of skin of theuser, a blood glucose level, blood oxygenation, and blood pressure. 5.The system of claim 2, wherein the ambient data is associated with atleast one of the following: light, heat, motion, moisture, and pressure.6. They system of claim 1, wherein the at least one sensor includes abiological sensor.
 7. The system of claim 1, wherein the customizedoutput data includes at least one of an audio output and a graphicsoutput.
 8. The system of claim 1, wherein the at least one computingresource includes at least one of the following: an applicationprogramming interface, a server, a cloud computing resource, a database,a network, and a blockchain.
 9. The system of claim 1, wherein the atleast one computing resource includes one of the following: asmartphone, a tablet computer, a phablet computer, a laptop computer, adesktop computer, an augmented reality device, a virtual reality device,a mixed reality device, a retinal implant, an artificial olfactiondevice, headphones, and an audio output device.
 10. The system of claim1, wherein the at least one computing resource includes one of a centralprocessing unit, a graphics processing unit, and a neural processingunit.
 11. The system of claim 1, wherein the at least one sensor isaffixed to a user.
 12. The system of claim 1, wherein the at least onesensor includes at least one of the following: a thermal imaging camera,a digital camera, a breath sensor, a depth sensor, a radar sensor, and agyroscope.
 13. The system of claim 1, wherein the at least one machinelearning technique includes one or more of the following: an artificialneural network, a convolutional neural network, a Bayesian neuralnetwork, a supervised machine learning algorithm, a semi-supervisedmachine learning algorithm, an unsupervised machine learning algorithm,and a reinforcement learning.
 14. The system of claim 1, wherein theautomatic and immediate personalized change in the user data includes atleast one of the following: a change of perception time, a change of arespiratory rate, a change of a breathing rate, a change of a heartrate, a change of a heart rate variability, a change of a hapticinteraction, a change of an electroencephalographic signal, a change ofan electrocardiographic signal, a change of an electromyographic signal,a change of a mechanomyographic signal, a change of an electrodermalactivity, a change of a motion, a change of a gesture, a change of apupil movement, a change of a biological structure, a change of amicroorganism, a change of a color of skin of the user, a change ofblood glucose levels, a change of a blood oxygenation, a change of ablood pressure, a change of a biological analyte, and change of a stresslevel.
 15. A method for customizing an output based on user data, themethod comprising: continuously capturing, by at least one sensor, theuser data associated with a user during perception of output data by theuser; analyzing, by at least one computing resource comprising a firstprocessor and a first memory, the user data received from the at leastone sensor; based on the analysis, determining dependencies between theuser data and the output data; and continuously customizing, by anadaptive interface comprising a second processor and a second memory,the output data using at least one machine learning technique based onthe analysis of the user data and the dependencies, the customizedoutput data intended to elicit an automatic and immediate personalizedchange of a biological response of the user to perception of thecustomized output data by the user without requiring the user to take anaction in response to the customized output data.
 16. The method ofclaim 15, further comprising: aggregating, by the at least one computingresource, further user data associated with a plurality of users intofederated user data; analyzing, by the at least one computing resource,the federated user data using collaborative machine learning; andadapting, by the at least one computing resource, the at least onemachine learning technique for individual users based on the results ofthe analysis of the federated user data.
 17. The method of claim 15,further comprising: continuously adapting, by the adaptive interface, amedia output based on user interactions with the adaptive interface. 18.The method of claim 15, wherein the at least one sensor includes adevice for analyzing electronic signals emitted by a user, wherein themethod further comprises extracting, by the device, one of aphysiological parameter of the user and an activity associated with theuser.
 19. The method of claim 15, wherein the continuous customizing ofthe output data includes at least one of the following: changing acolor, playing audio-perceived stimuli, providing a haptic feedback,changing a font, changing a shape of the font, changing a brightness,changing a contrast, changing an illuminance, changing a warmth,changing a saturation, changing a fade, changing a shadow, changing asharpness, changing a structure, generating computer images, changing abass, changing a volume, changing a pitch of a sound, changing a treble,changing a balance, changing a graphical user interface, and changing auser experience design.
 20. A machine learning system for customizingoutput based on user data, the system comprising: at least one sensorconfigured to continuously capture the user data associated with a userduring perception of output data by the user; at least one computingresource comprising a first processor and a first memory, the at leastone computing resource being configured to: analyze the user datareceived from the at least one sensor; based on the analysis, determinedependencies between the user data and the output data; and aggregatefurther user data associated with a plurality of users into federateduser data; analyze the federated user data using collaborative machinelearning; and adapt the at least one machine learning technique forindividual users based on the results of the analysis of the federateduser data; and an adaptive interface comprising a second processor and asecond memory, the adaptive interface being configured to: continuouslycustomize the output data using at least one machine learning techniquebased on the analysis of the user data and the dependencies, thecustomized output data intended to elicit an automatic and immediatepersonalized change of a biological response of the user to perceptionof the customized output data by the user without requiring the user totake an action in response to the customized output data; andcontinuously adapt a media output based on user interactions with theadaptive interface.